Feature Selection and Non-linear Feature Extraction

Feature extraction and feature selection are two important tasks in pattern recognition. Classiication algorithms like k-nearest neighbors, which are based on the assumption that patterns in the same class are close to each other and those in diierent classes are far apart (locality property), rely heavily on the quality of the features extracted from the input data. In this work, an objective function, which translates the locality property into a linear, Fisher like criteria, based on within and between class variance of the training data is proposed. This criteria is used for the two tasks of feature selection and feature extraction. Feature selection is done by introducing relevance measures for each input dimension. Feature extraction is deened as a linear combination of a set of non-linear functions. Closed form solutions for both, the relevance measures in feature selection and for the weights of linear combinations for feature extraction task are derived using the Fisher like criteria. Experiments over synthetic and real datasets have been used to highlight the strengths and weaknesses of these methods. Feature selection improves the performance of k-nearest neighbor classiier signiicantly for both synthetic and real data sets, while the feature extractor is found to be able to extract features from input spaces which are not suitable for simple classiiers like linear discriminant and k-nearest neighbors.